AI poisoning a rising concern among regional tech leaders

In the evolving business landscape, artificial intelligence (AI) is being widely used, but companies are still unaware of the challenges associated with it, analysts say. 

Currently, ‘AI data/model poisoning,’ has emerged as a critical challenge, which occurs when malicious actors deliberately manipulate an AI or machine learning model’s training data, compromising its reliability. 

Mostly, this attack exploits the predictive or narrow AI solutions (task-focused AI systems) within the MLOps cycle (deploying machine learning models). Within generative AI solutions this poisoning appears in RAG (answers using retrieved knowledge) and knowledge graphs (connected data for understanding) rather than at the model levels. Most concerning is impact on the Agentic AI, where AI poisoning not only corrupts the output but also influences autonomous actions. 

For instance, consider an oil and gas company is using AI from predictive maintenance. Here, if a malicious actor subtly injects manipulated sensor readings into its training data, the AI might fail identify genuine warning signs or even falsely flag healthy equipment. This could lead to unexpected shutdowns, costly repairs and potential safety hazards.

Similarly, in the financial sector, corrupted stock data in a bank’s investment agent’s knowledge graph could prompt the agentic AI to make poor investment choices, resulting significant losses for customers. “AI poisoning undermines the reliability of AI-driven decisions, making companies vulnerable to operational disruptions and financial losses. This highlights the significance of protecting the integrity of data to ensure the continued safe and efficient operation of assets,” Premchand Kurup, CEO of Paramount, told Khaleej Times in an interview.

Data poisoning presents a grave threat to enterprises, particularly in sensitive sectors such as finance and cybersecurity. For instance, when attackers corrupt training data in a bank’s AI-powered fraud detection system, it may fail to identify real fraud, causing significant financial losses. Similarly, in cybersecurity, poisoned malware detection system might misclassify threats as safe, leaving systems vulnerable to attack. The consequences could extend beyond immediate loses, as data poisoning can erode customer trust and cause reputational harm. “Detecting these sophisticated attacks requires robust AI cybersecurity framework and protection mechanisms. An inefficiency in these resources may lower organisations’ trust in AI initiatives,” Kurup said.

As AI integration accelerates companies need to adopt a comprehensive AI Framework for Cybersecurity to ensure that the technology is implemented safely and responsibly, Kurup said. “The first component of this framework is AI governance, which establishes clear guidelines for responsible AI usage, addressing data privacy concerns and legal liabilities while boosting productivity. Second, securing AI systems to protect AI models from external threats through a comprehensive AI lifecycle approach, from data collection to deployment and retirement. This ensures AI integrity and resilience by reducing the risks of system exploitation,” Kurup said.

Premchand Kurup, CEO of Paramount

The third factor is the use of AI in cybersecurity. For instance, integrating AI in Security Operations Centres (SOCs) to enhance threat detection and response capabilities. Similarly, using Agentic AI in Identity and Access Management (IAM) to manage user access and proactive risk mitigation. Finally, Data and Integration security to protect data as it moves between different systems by preventing unauthorised access. “At Paramount, we are committed to delivering an effective framework across key cybersecurity domains, including Network Security, Identity and Access Management, Cloud Security, Data Security, AI in Cybersecurity and AI Adoption Framework,” Kurup said. 

While no specific AI poisoning incidents have been reported in the GCC region, the evolving technological transformations signifies surging cybersecurity awareness. The region has evolved from following global trends to becoming a proactive early adopter of AI technologies. 

Generative AI alone are projected to contribute $21 billion to $35 billion annually to the GCC economies, adding to the $150 billion from other AI technologies. This represents 1.7 per cent to 2.8 per cent of the GCC’s current annual non-oil GDP. A recent McKinsey survey indicates that nearly three-quarters of respondents reported their organisations were already utilising generative AI in at least one business function, with over half of the GCC respondents investing at least 5 per cent of their digital budgets in generative AI, higher than the global average of 33 per cent. Despite impressive adoption, reports on AI threats are not widely available, due to the early stage of adoption and potential concerns around reputational damage. 

However, the GCC region’s position as a global economic powerhouse, coupled with prevailing geopolitical concerns, necessitates vigilance. “With global technology leaders, including Apple and Amazon, having experienced AI poisoning, the region must implement robust security measures to safeguard AI systems and their substantial economic contributions,” Kurup said. 

AI poisoning impacts more than financial status, as it can jeopardise a company’s brand reputation, with varying severity across sectors. For example, in an oil and gas company, a poisoned predictive maintenance system failing to detect equipment failure can lead to costly shutdowns and repairs. In the energy sector, downtime can cost around $2.48 million per hour. Fortune Global 500 companies face average annual unplanned downtime cost of around $129 million per facility.

“Though quantifying the exact financial losses remains difficult due to AI’s evolving nature AI, the potential for critical economic damage is undeniable. As AI becomes more deeply integrated into critical infrastructure and business processes, the financial implications of successful AI poisoning attacks are expected to intensify,” Kurup said.